Skip to content

Improve handling of different variants. - #736

Open
rhubert wants to merge 10 commits into
BobBuildTool:masterfrom
rhubert:show-recursive-diff
Open

rhubert wants to merge 10 commits into
BobBuildTool:masterfrom
rhubert:show-recursive-diff

Conversation

@rhubert

@rhubert rhubert commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Debugging:

  • "incompatible variants error"
  • "why do I have two variants of foo?"

is tricky if the difference orients from a deeply nested package. Improve this by adding a recursive variant diff.

@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.07359% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.40%. Comparing base (e05a6d5) to head (f3232db).

Files with missing lines Patch % Lines
pym/bob/pkgdiff.py 92.81% 13 Missing ⚠️
pym/bob/input/recipe.py 88.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #736      +/-   ##
==========================================
+ Coverage   90.39%   90.40%   +0.01%     
==========================================
  Files          59       60       +1     
  Lines       17408    17536     +128     
==========================================
+ Hits        15736    15854     +118     
- Misses       1672     1682      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rhubert
rhubert force-pushed the show-recursive-diff branch 2 times, most recently from 6604d25 to 9be718d Compare November 9, 2026 13:24
@rhubert
rhubert marked this pull request as draft November 9, 2026 14:21
@rhubert
rhubert force-pushed the show-recursive-diff branch from 9be718d to b555718 Compare November 9, 2026 14:53
@rhubert rhubert changed the title Show recursive diff Improve handling of different variants. Nov 9, 2026
@rhubert
rhubert marked this pull request as ready for review November 9, 2026 14:56
@jkloetzke

Copy link
Copy Markdown
Member

Good idea to improve the error message there. It can be indeed hard to see why exactly the versions differ.

I made a small test with the linux-blueprint by doing this:

diff --git a/recipes/rootfs.yaml b/recipes/rootfs.yaml
index f70530f..2f649a8 100644
--- a/recipes/rootfs.yaml
+++ b/recipes/rootfs.yaml
@@ -30,6 +30,7 @@ depends:
       use: [tools]
       depends:
           - graphics::xorg::mkfontscale
+          - libs::openssl-tgt
 
     - core::rootfs-skel
     - core::dbus

So I've explicitly depended on libs::openssl-tgt build with the host-compat-toolchain but many packages will provide the linux cross compiled version. Then I get this output:

$ bob ls
Parse error: Incompatible variants of package: /amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt vs. /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt
Processing stack: amd64-glibc-busybox-x11/rootfs
This error is caused by 'libs::openssl-tgt' that is passed upwards via 'provideDeps' from multiple dependencies of 'rootfs'.
These dependencies constitute different variants of 'libs::openssl-tgt' and can therefore not be used in 'rootfs'.
Concrete differences:
package: /amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt/dist (e379386e2ab918e157ab5ba7d65874b6c2684dae) <> /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt/dist (0f47334f938090d2de3bdc3404a059995f78c916)
- environment.AR=x86_64-cross-linux-gnu-ar
+ environment.AR=x86_64-bob_compat-linux-gnu-ar
package: /amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt/dist (e379386e2ab918e157ab5ba7d65874b6c2684dae) <> /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt/dist (0f47334f938090d2de3bdc3404a059995f78c916)
- environment.AUTOCONF_HOST=x86_64-cross-linux-gnu
+ environment.AUTOCONF_HOST=x86_64-bob_compat-linux-gnu
package: /amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt/dist (e379386e2ab918e157ab5ba7d65874b6c2684dae) <> /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt/dist (0f47334f938090d2de3bdc3404a059995f78c916)
- environment.CC=x86_64-cross-linux-gnu-gcc
+ environment.CC=x86_64-bob_compat-linux-gnu-gcc
package: /amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt/dist (e379386e2ab918e157ab5ba7d65874b6c2684dae) <> /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt/dist (0f47334f938090d2de3bdc3404a059995f78c916)
- environment.CFLAGS=-O2 -g -pipe
+ environment.CFLAGS=-O2 -pipe -fPIC
package: /amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt/dist (e379386e2ab918e157ab5ba7d65874b6c2684dae) <> /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt/dist (0f47334f938090d2de3bdc3404a059995f78c916)
- environment.CPPFLAGS=-Wdate-time
+ environment.CPPFLAGS=
...

I'm afraid that is next to unreadable. Can we have only one header for the package and have all differences in a package next to each other. Also the SHA1 sum is not really helpful. I guess it's the variant-ID, is it? Unless there is a deeper reason, I wouldn't show it.

The /dist suffix is misleading because there is no package dist. Was there a specific reason to use a different format than bob show --format diff? If possible, I would like to keep things consistent.

Is there a limit how deep the recursion goes? In the worst case, recursing the whole dependency tree is not viable. My gut feeling is that recursion should happen only if the package that is currently compared only differs on dependencies. In other words, like above, if there are already differences in variables the recursion should stop. There is no need to look deeper.

Likewise, if the dependency differs, it should only be followed if the dependency is derived from the same recipe. Imagine the variants are caused by some conditional dependencies. The fact that there is a new dependency (or has been removed) is enough.

But in principle, I really like the idea. I haven't looked at the implementation. I think we first have to get the output minimized...

@rhubert
rhubert force-pushed the show-recursive-diff branch 2 times, most recently from 14a7cfb to 0b88b4a Compare November 10, 2026 08:12
@rhubert

rhubert commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the input. The output now reuses the existing diff format and shows the diff of the first package which differs. For your example:

❯ bob ls
Parse error: Incompatible variants of package: /amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt vs. /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt
Processing stack: amd64-glibc-busybox-x11/rootfs
This error is caused by 'libs::openssl-tgt' that is passed upwards via 'provideDeps' from multiple dependencies of 'rootfs'.
These dependencies constitute different variants of 'libs::openssl-tgt' and can therefore not be used in 'rootfs'.
Concrete differences:
/amd64-glibc-busybox-x11/rootfs/graphics::xorg::xserver/libs::openssl-tgt <> /amd64-glibc-busybox-x11/rootfs/libs::openssl-tgt
- buildTools.target-toolchain=devel::cross-toolchain-x86_64-linux-gnu
+ buildTools.target-toolchain=devel::host-compat-toolchain
- buildVars.AR=x86_64-cross-linux-gnu-ar
+ buildVars.AR=x86_64-bob_compat-linux-gnu-ar
- buildVars.AUTOCONF_HOST=x86_64-cross-linux-gnu
+ buildVars.AUTOCONF_HOST=x86_64-bob_compat-linux-gnu
...

In my tree the issue I tried to understand was that the output of bob ls -d //devel::host-compat-toolchain | wc -l was 2 and the existing bob show --format diff

❯ bob show --format diff //devel::host-compat-toolchain
- buildTools.target-toolchain=ada::tests::buildall-aarch64/devel::host-compat-toolchain/devel::compat::cross-toolchain
+ buildTools.target-toolchain=arm64-qemu-virt-multicore/devel::host-compat-toolchain/devel::compat::cross-toolchain
- depends[0]=ada::tests::buildall-aarch64/devel::host-compat-toolchain/devel::compat::binutils
+ depends[0]=arm64-qemu-virt-multicore/devel::host-compat-toolchain/devel::compat::binutils
- depends[1]=ada::tests::buildall-aarch64/devel::host-compat-toolchain/devel::compat::gcc-canadian-cross
+ depends[1]=arm64-qemu-virt-multicore/devel::host-compat-toolchain/devel::compat::gcc-canadian-cross
- meta.buildVariantId=10c4757bc31f84bed573beea37e9c1c42a5bd665
+ meta.buildVariantId=799eb3354454fca61ed94e1c336922c05b334cee
- meta.package=ada::tests::buildall-aarch64/devel::host-compat-toolchain
- meta.packageVariantId=f22518aee1a96f967150ecff62cc8aa9b64e954b
+ meta.package=arm64-qemu-virt-multicore/devel::host-compat-toolchain
+ meta.packageVariantId=a6c057918a8a4018c4f817b192c082b944ebda3f
- packageTools.target-toolchain=ada::tests::buildall-aarch64/devel::host-compat-toolchain/devel::compat::cross-toolchain
+ packageTools.target-toolchain=arm64-qemu-virt-multicore/devel::host-compat-toolchain/devel::compat::cross-toolchain

wasn't helpful as it only confirmed both variants are using different tools. Now it shows:

❯ bob show --format diff --recursive //devel::host-compat-toolchain
ada::tests::buildall-aarch64/devel::host-compat-toolchain/perl::perl <> arm64-qemu-virt-multicore/devel::host-compat-toolchain/perl::perl
- meta.package=ada::tests::buildall-aarch64/devel::host-compat-toolchain/perl::perl
- meta.packageVariantId=8b902c0f85b4cf0a8363926d1d2d3d6344b34024
+ meta.package=arm64-qemu-virt-multicore/devel::host-compat-toolchain/perl::perl
+ meta.packageVariantId=31ef4d668c588653c9539f1d49e2269c6862567d
- packageVars.PERL_ARCH=x86_64-linux
+ packageVars.PERL_ARCH=aarch64-linux

pointing me to the package built with a cross-target-toolchain while inheriting basement::rootrecipe ...

Also improved the diff output by suppressing dependency differences if they have the same variantId.

@jkloetzke

Copy link
Copy Markdown
Member

Thanks. This is already looking much better. I created a small project to test the feature:
test-recursive-diff.zip

If I just do a bob ls, the following output is shown:

test-recursive-diff$ bob ls
Parse error: Incompatible variants of package: /root/l1-b/l2 vs. /root/l1-a/l2
Processing stack: root
This error is caused by 'l2' that is passed upwards via 'provideDeps' from multiple dependencies of 'root'.
These dependencies constitute different variants of 'l2' and can therefore not be used in 'root'.
Concrete differences:
l3 <> l3
- meta.packageVariantId=975f054787b7bb6f0fe0e4ee33a8fee5f533214c
+ meta.packageVariantId=0fb28c327f32928bf155e6e25cbf153a1623247a
- packageVars.L3=second
+ packageVars.L3=first

It shows correctly l3 as the culprit here because it provided a different tool variant. But there is no path shown. If the problem is a bit more involved, the user would be confused because there is no /l3 package. Intuitively, I would have expected:

Concrete differences:
/root/l3 <> /root/l1-b/l3
...

If you comment out the provideDeps in l1.yaml, you can actually see it:

$ bob show --format diff --recursive root/l1-a/l2 root/l1-b/l2
root/l3 <> root/l1-b/l3
- meta.package=root/l3
+ meta.package=root/l1-b/l3
- meta.packageVariantId=0fb28c327f32928bf155e6e25cbf153a1623247a
+ meta.packageVariantId=975f054787b7bb6f0fe0e4ee33a8fee5f533214c
- packageVars.L3=first
+ packageVars.L3=second

What is missing IMHO in both cases, is how we got there. In principle, this is something like an exception trace. How about something like the following?

$ bob show --format diff --recursive root/l1-a/l2 root/l1-b/l2
diff root/l1-a/l2 root/l1-b/l2
--- root/l1-a/l2
+++ root/l1-b/l2
- buildTools.asdf=root/l3
+ buildTools.asdf=root/l1-b/l3
- meta.buildVariantId=60d63115983d91f4ca3fcdbf0ad362625eb718b4
+ meta.buildVariantId=50d9b6029dc91e83b00079d06d5a072a7f941036
- meta.package=root/l1-a/l2
+ meta.package=root/l1-b/l2
- meta.packageVariantId=267007b2179c5f2a39512f9653faf17f049b5076
+ meta.packageVariantId=3738715b44e0beda5eb7275f1e2c1ee65ff6a94c
- packageTools.asdf=root/l3
+ packageTools.asdf=root/l1-b/l3

The above package diff was caused by 'buildTools.asdf':

diff root/l3 root/l1-b/l3
--- root/l3
+++ root/l1-b/l3
- meta.package=root/l3
+ meta.package=root/l1-b/l3
- meta.packageVariantId=0fb28c327f32928bf155e6e25cbf153a1623247a
+ meta.packageVariantId=975f054787b7bb6f0fe0e4ee33a8fee5f533214c
- packageVars.L3=first
+ packageVars.L3=second

@rhubert
rhubert force-pushed the show-recursive-diff branch 3 times, most recently from 99be084 to 3b4200e Compare November 12, 2026 07:26
@rhubert

rhubert commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

I really like the exception trace idea. With this the user can follow the error back to it's root. 👍

With multiple_host_toolchains.diff.gz to the linux-blueprint bob show --format diff --recursive //devel::host-compat-toolchain now shows the chain of 4 packages leading to the different variants.

@jkloetzke jkloetzke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor naming thing. Overall, I'm very pleased with the feature. 👍

Comment thread pym/bob/pkgdiff.py Resolved
Comment thread pym/bob/variant.py Resolved
Comment thread pym/bob/pkgdiff.py Outdated
"""Find the first package with direct differences in variant inputs."""
visited = set()

def compare(a, b, recipeOnly=False):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does recipeOnly mean exactly? And why do we have to filter the fingerprint parts of the variantId? I'm afraid I don't see it (yet).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was not to blame a tool if only the fingerprint script of the tool or it's deps differs. However, I'm unable to build a concrete example for this setup, so I guess this can be removed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a deeper look again and it does have its merit. In the variant-id calculation the fingerprint part of tools is ignored. If the "recipe" part of a tool does differ, the recursion needs to ignore the fingerprint parts.

I pushed some fixups that fully align the calcPackageDiffRecursive with CoreStep.getDigest(). Interestingly, Claude noted that tools that differ only in the path or libs are now correctly detected but were not visible in the package diff. The last commit fixes this.

Comment thread pym/bob/variant.py Resolved
Comment thread pym/bob/variant.py Resolved
Comment thread pym/bob/variant.py Resolved
Comment thread pym/bob/input/recipe.py Resolved
Comment thread pym/bob/cmds/show.py Resolved
@jkloetzke

Copy link
Copy Markdown
Member

I added a small fixup that defers the calculation of the packages until they are actually needed. Please have a look and squash if you think this is good enough.

@rhubert
rhubert force-pushed the show-recursive-diff branch from dd17066 to e42042f Compare November 15, 2026 17:41
Share package inspection and diff rendering without changing their
behavior. This prepares the helpers for use while reporting incompatible
variants.
Trace differing dependencies to the first concrete package
difference and show the comparisons that led there. Resolve
provider references with their ambient context so diagnostic
paths remain accurate.
Check that conflicting variants report their differences and retain
provider paths through inherited tools.
Follow differing dependencies to the first concrete package difference
and show the chain of package comparisons. Reuse the regular diff output
and its field selection and common-value options.
Document how recursive comparisons follow dependencies and apply the
selected diff options.
Exercise recursive comparisons, dependency equivalence and the supported
diff display options.
@rhubert
rhubert force-pushed the show-recursive-diff branch from e42042f to c0e7885 Compare November 15, 2026 18:54
The path and library paths of tools are part of the Variant-Id. Show
them too, so that variant differences caused by them are visible in
the (recursive) package diff. Tools are now shown as map with the
providing package, path and libs instead of just the package path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants